The present invention relates to a face-image processing apparatus for determining opening/closing of the eyes in accordance with the multivalue face image of a person to be detected.
As a conventional face-image processing apparatus incorporating an illuminating means for irradiating the face of a person to be detected, an apparatus disclosed in the Unexamined Japanese Patent Application Publication No. Hei.9-198508 has been known.
FIG. 7 is a schematic view showing the structure of an conventional eye-state detecting apparatus disclosed in the Unexamined Japanese Patent Application Publication No. Hei.9-198508.
Referring to FIG. 7, reference numeral 1 represents a person to be detected, 2 represents a CCD camera for photographing the person 1 to be detected and 3 represents a multivalue image memory for storing the image photographed by the camera 2. Reference numeral 4 represents a binarizing means for binarizing the multivalue image, 5 represents a binarized-image memory for storing the binarized image, 6 represents a binarized-image characteristic extracting means for extracting the characteristic from the binarized image and 7 represents an opening/closing determining means for determining opening/closing of the eyes in accordance with the characteristic of the binarized image. Reference numeral 8 represents an LED light source and 9 represents an LED operating circuit for passing an electric current to the LED light source 8.
FIGS. 8A and 8B show multivalue image of the face of a person to be detected and a binarized image reflected by the retina obtained by the apparatus disclosed in the Unexamined Japanese Patent Application Publication No. Hei.9-198508. FIG. 8A shows the multivalue image. FIG. 8B shows the binarized image.
Referring to FIGS. 8A and 8B, reference numeral 10 represents the multivalue image, 11 represents the binarized image, 14 represents the image reflected by the retina and 15 represents a binarized region of the image reflected by the retina.
When the face of the person 1 to be detected is photographed by the foregoing conventional face-image processing apparatus, the pupils of the person 1 to be detected are photographed as if the pupils glow owning to light reflected by the retinas of the eyeballs, as shown in FIG. 8A. The reason for this lies in that the retinas have a characteristic which returns reflected light in the same direction as the direction of the incident light. The image 14 reflected by the retina is photographed with considerably intense brightness as compared with the other portions of the face. Therefore, a process for binarizing the overall portion of the photographed multivalue image enables a region having the intense brightness as shown in FIG. 8B to be extracted as a pupil region. In accordance with a fact whether or not the pupil region exists, that is, whether or not the binarized region 15 of the image reflected by the retina exists, the opening/closing state of the eyes is determined.
FIG. 9 shows a flow chart of an algorithm employed by the face-image processing apparatus disclosed in the Unexamined Japanese Patent Application Publication No. Hei.9-198508 for extracting the eyes.
An image of the face photographed in step S1 is stored in the multivalue-image memory for one frame in step S2. A binarizing process is performed in step S3 so that a multivalue image captured in step S2 and output from the binarized-image memory is binarized by using a threshold value for binarization. A result of the binarizing process performed in step S3 is stored in the binarized-image memory for one frame in step S4. In step S5 an eye-candidate binarized region setting process is performed so that an eye candidate region is set from the binarized regions output from the binarized-image memory. In step S6 the binarized eye region is selected from the eye candidate region set in step S5. If the binarized eye region is selected in step S6, opening/closing is determined in step S7 such that an image reflected by the retina exists. That is, a determination is made that the eyes are opened. If the binarized eye region is not selected in step S6, opening/closing is determined in step S7 such that no image reflected by the retina exists. That is, the eyes are closed.
As a conventional face-image processing apparatus using a template, an apparatus disclosed in, for example, the Unexamined Japanese Patent Application Publication No.
Hei.8-175218 is known. According to the foregoing disclosure, a driving state detecting apparatus has been disclosed which incorporates a template producing means for vertically and laterally moving a predetermined standard face template with respect to the photographed image to perform correlation calculations so as to produce an object template for the person to be detected; and an eye-region detecting means for detecting the eye region of the person to be detected by performing correlation calculations by using the object template.
FIG. 10 is a schematic view showing the structure of the driving state detecting apparatus disclosed in the Unexamined Japanese Patent Application Publication No. Hei.8-175218.
An image processing apparatus 17 is connected to a camera 2 for photographing a person 1 to be detected. An image of the face of the person 1 to be detected is supplied to the image processing apparatus 17. The image processing apparatus 17 incorporates an A/D converter, a normalizing circuit and a correlation calculating circuit so as to convert the supplied image signal into a digital signal, followed by performing a process for normalizing light and shade. A memory 18 is connected to the image processing apparatus 17. A standard template and layout data of elements of the face including the eyes and eyebrows are previously stored in the memory 18. Moreover, an electronic control unit ,i.e., ECU 19 is connected to the image processing apparatus 17 so that a result of the process is supplied to the ECU 19. The ECU 19 determines a state of driving of the person 1 to be detected in accordance with the result of the process. Thus, the ECU 19 outputs a control signal to an alarm unit 20 so that an alarm is issued.
As a conventional face-image processing apparatus using distribution of brightness, an apparatus disclosed in, for example, the Unexamined Japanese Patent Application Publication No. Hei.6-348980 is known. The foregoing apparatus is an apparatus which is capable of monitoring the eyes of a driver in a vehicle and structured to specify the positions of the eyes by paying attention to a fact that the black eye portions are the darkest portions in the frame. An average value of maximum and minimum brightness in the vicinity of the darkest point is used as the threshold value of the brightness so as to extract the eye region.
Specifically, a plurality of sampling points are determined at the positions of the eyes on the frame, and then each sampling point is shifted to the dark portion. In accordance with the sampling points converged to the black eye portions, the position of the eyes are specified. In accordance with the distribution of the brightness of the adjacent portions, the eye regions are extracted.
FIG. 11 is a diagram showing a process for extracting the eye regions which is performed by the apparatus disclosed in the Unexamined Japanese Patent Application Publication No. Hei.6-348980 and which uses the distribution of the brightness so as to monitor the eyes of the driver in the vehicle.
FIG. 11 shows a method with which mean brightness T_av of maximum brightness T (max) of portions adjacent to the eye and minimum brightness T (min) of the same is used as a threshold value so as to extract a black eye portion so that the width W of the eye is obtained.
The conventional apparatus which detects the state of the eyes by using an image reflected by the retinas incorporates the binarized-image characteristic extracting means 6 which binarizes a multivalue image so that the image process is performed. Therefore, information about the light and shade of the multivalue image has been abolished. Hence it follows that extraction of images reflected by the retinas requires control of the threshold value for the binarization in accordance with the brightness of the photographed image of the face of the person to be detected. The result of the extraction of the characteristic greatly depends on the foregoing threshold value for the binarization.
FIGS. 12A and 12B show conventional binarized image of the person to be detected including an image reflected by the retina. FIG. 12A shows a result of binarization of only the image reflected by the retina. FIG. 12B shows an image of a binarized image including noise.
Referring to FIGS. 12A and 12B, reference numerals 11 and 15 represent the same elements as those shown in FIGS. 8A and 8B. Reference numeral 21 represents noise.
FIG. 13 shows the distance between the two eyes and the levelness of a multivalue image of the face of a person to be detected by the conventional apparatus.
Referring to FIG. 13, reference numerals 1 and 10 represent the same elements as those shown in FIGS. 8A and 8B.
For example, the overall pupils of the multivalue image of the face are binarized with a general threshold for the binarization. Therefore, if a dark image is reflected by the retina or if the overall portion of the fame is a bright image, that is, if the image has low contrast, extraction of the image reflected by the retina as a binarized region has sometimes been inhibited.
It is preferable that the binarized image is processed such that only the image reflected by the retina is extracted as the binarized region, as shown in FIG. 12A. If a region having a brightness band which is the same as that of the image reflected by the retina exists, the region, however, is binarized similarly to the image reflected by the retina. Thus, the region is left as noise 21, as shown in FIG. 12B. As a result, the extraction of the image reflected by the retina from a plurality of the binarized regions must be performed in accordance with relative positional relationships including the distance and levelness with which the conditions for the two eyes are satisfied. If the face is inclined, the levelness of the foregoing positional relationships cannot be satisfied, as shown in FIG. 13. As a result, an error extraction is caused.
If the face is moved vertically, the positional relationship between the eyes and the eyebrows is not changed. Therefore, when regions adjacent to the eyes have been detected by using the regions adjacent to the object eyes, the conventional apparatus for detecting a state of driving using the template is able to reliably specify the eye region such that upper black portions are determined as the eyebrows and the lower black portions are determined as the eyes. When the movement of the face is factored in, matching of the template must be performed by scanning a large area of the image. As a result, time required to complete the process of calculating correlation cannot be shortened.
The conventional apparatus for monitoring the eyes of a driver in a vehicle arranged to use distribution of brightness is able to perform extraction of the eye regions such that the eye region can reliably be included in the sampling point of the darkest point. Therefore, a multiplicity of the sampling points must be provided. As a result, black regions, such as eyebrows and hairs, are included. Since the multiplicity of the points of interest are processed, time required to complete the process cannot be shortened.
To shorten time required to complete the process, a temperature measuring device TMD has been employed as a means for limiting the face image region of the person to be detected. Hence it follows that the structure of the system becomes too complicated and the cost cannot be reduced.
In view of the foregoing, an object of the present invention is to provide a face-image processing apparatus which is capable of reliably determining opening/closing of the eyes.
A face-image processing apparatus according to the present invention comprises: image input means for inputting a multivalue face image which is expressed by a multiplicity of pixels disposed in a matrix configuration; multivalue-image characteristic extracting means for extracting a binarized eye region from the multivalue face image input by the image input means; and opening/closing determining means for determining opening/closing of the eyes in accordance with the eye region extracted by the multivalue-image characteristic extracting means.
The apparatus comprises: image input means for inputting a multivalue face image which is expressed by a multiplicity of pixels disposed in a matrix configuration; multivalue-image characteristic extracting means for extracting an eye region from the multivalue face image input by the image input means; and opening/closing determining means for determining opening/closing of the eyes in accordance with the eye region extracted by the multivalue-image characteristic extracting means.
The multivalue-image characteristic extracting means extracts a pixel of interest in accordance with distribution of relative brightness of the brightness of the pixel of interest contained in the multivalue face image and the brightness of pixels adjacent to the pixel of interest so as to extract an eye region by using the extracted pixel of interest.
The apparatus further comprises extracted-image storing means for storing the extracted pixel of interest, wherein the extraction of the eye region is performed by the multivalue-image characteristic extracting means by using the pixel of interest stored in the extracted-image storing means.
The extraction of the pixel of interest is performed by the multivalue-image characteristic extracting means in accordance with distribution of relative brightness in one direction.
The extraction of the pixel of interest in accordance with the distribution of the relative brightness is performed by dedicated hardware.
The distribution of the relative brightness in one direction is in the horizontal direction of the face.
The extraction of the pixel of interest in accordance with the distribution of the relative brightness is performed by dedicated hardware.
The eye region includes an image reflected by the retina.